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Abstract 

As a main step in the numerical solution of control problems in continuous time, 
the controlled process is approximated by sequences of controlled Markov chains, thus 
discretising time and space. A new feature in this context is to allow for delay in the 
dynamics. The existence of an optimal strategy with respect to the cost functional 
can be guaranteed in the class of relaxed controls. Weak convergence of the approxi- 
mating extended Markov chains to the original process together with convergence of 
the associated optimal strategies is established. 



1 Introduction 

A general strategy for rendering control problems in continuous time accessible to numerical 
computation is the following: Taking as a starting point the original dynamics, construct 
a family of control problems in discrete time with discrete state space and discretised cost 
functional. Standard numerical schemes can be applied to find an optimal control and to 
calculate the minimal costs for each of the discrete control problems. The important point 
to establish is then whether the discrete optimal controls and minimal costs converge to 
the continuous-time limit as the mesh size of the discretisation tends to zero. If that is the 
case, then the discrete control problems are a valid approximation to the original problem. 
Approximation schemes for non-delay stochastic control problems in continuous time 



i mplem enting the general strategy just outlined are well established, see lKushner and Dupui 



( 2nnih . T he meth o d yie lds convergence results under very general conditions. In the non- 
delay case Krvlov (|2nnnl ) derived rates of convergence for those schemes by exploiting fine 



analytical properties of the associated Bellman equations. 

* Financial support by the DFG-Sonderforschungsbereich 649 Economic Risk is gratefully acknowledged. 
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The dynamics of the control problem we are interested in are described by a stochastic 
delay differential equation (SDDE). Thus, the future evolution of the dynamics may depend 
not only on the present state, but also on the p ast evolut i on. F or an exposition of the 



general theory of SDDEs see iMohammedl l|l984r i or 



Mao 



Th e development o f 



numerica l meth ods for SDDEs has attracted much attention recently , see lBuckwan l|2nnfll ). 



Hu et alJ I|2nn4l ) and the references therein. In lCalzolari et all ((2il0^ a rate of convergence 



for a segmentwise Euler scheme is obtained (Proposition 4.2 ibid.) and is used in a non- 
linear filtering problem for approximating the state process, which is given by an SDDE. 
Numerical procedures fo r deterministic control w ith delayed dynamics have already been 
used in applications, see lBoucekkine et all l|2r)fl5r i for the analysis of an economic growth 
model. The algorithm proposed there is based on the discretisation method studied here, 
but no formal proof of convergence is given. 

The mathematical analysis of stochastic control pro blems with time delay in the state 
equation has been the objec t of recent works, se e e.g. 



Elsanosi et al 



I 2nnm for certain 

explicitly avai lable solutions , l0ksenda] and SulemI (|2nnil ) for the derivation of a maximum 
principle and iLarssenl (120021 ) for the dynamic programming approach. Although one can 
invoke the dynamic programming principle to derive a Hamilton-Jacobi-Bellman equation 
for the value function, such an equation will in general be a non-linear partial differential 
equation on a functional state space. The analytical methods for the non-delay case do not 
simply carry over to this infinite- dimensional setting. Another approach to treat stochastic 
control problems with delay i s based on repres e nting the state equation as an evolution 
equation in Hilbert space, see 



Bensoussan et al 



(iMa). 



The class of control problems is specified in Section O In Section |S] we prove the 
existence of optimal strategies for those problems in the class of relaxed controls. Section 
introduces the approximating processes and provides a tightness result. Finally, in Section^ 
the discrete control problems are defined and the convergence of the minimal costs and 
optimal strategies is shown. 



2 The control problem 

We consider the control of a dynamical system given by a one-dimensional stochastic 
delay differential equation (SDDE) driven by a Wiener process. Both drift and diffusion 
coefficient may depend on the solution's history a certain amount of time into the past. 
Let r > denote the delay length, i.e. the maximal length of dependence on the past. 
For simplicity, we restrict attention to the case, where only the drift term can be directly 
controlled. 

Typically, the solution process of an SDDE does not enjoy the Markov property, while 
the segment process associated with that solution does. For a real-valued cadlag function 
(i.e., right-continuous function with left-hand limits) i/j living on the time interval [— r, oo) 
the segment at time t £ [0, cxd) is defined to be the function ^pt ■ [—r, 0] ^ M given 
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by tptis) := '4'{t + s). Thus, the segment process {Xt)t>o associated with a real-valued 
cadlag process {X{t))t>-r takes its values in Dq:= D{[—r,0]), the space of all real- valued 
cadlag functions on the interval [— r, 0]. There are two natural topologies on Dq. The 
first is the one induced b y the supremum norm. The second is the Skorohod topology of 



cadlag convergence fe. g. iBillingslevl . Ilflflfjl l. The main difference between the Skorohod 
and the uniform topology lies in the different evaluation of convergence of functions with 
jumps, which appear naturally as initial segments and discretised processes. For continuous 
functions both topologies coincide. Similar statements hold for Dqo := D{[—r,<X))) and 
Doo := -D([0,oo)), the spaces of all real-valued cadlag functions on the intervals [— r, oo) 
and [0,00), respectively. The spaces D^o and D^o will always be supposed to carry the 
Skorohod topology, while Dq will canonically be equipped with the uniform topology. 

Let (r,(ir) be a compact metric space, the space of control actions. Denote by h 
the drift coefficient of the controlled dynamics, and by a the diffusion coefficient. Let 
{W{t))t>Q be a one-dimensional standard Wiener process on a filtered probability space 
(rj, {J-t)t>o,^) satisfying the usual conditions, and let {u{t))t>Q be a control process, i. e. 
an (.F()-adapted measurable process with values in F. Consider the controlled SDDE 

(1) dX{t) = b{Xt,u{t))dt + a{Xt)dW{t), t>0. 

The control process u{.) together with its stochastic basis including the Wiener process is 
called an admissible control if, for every deterministic initial condition if £ Dq, Eq. ^ has 
a unique solution which is also weakly unique. Write Uad for the set of admissible controls 
of Eq. . The stochastic basis coming with an admissible control will often be omitted 
in the notation. 

A solution in the sense used here is an adapted cadlag process defined on the stochastic 
basis of the control process such that the integral version of Eq. Q is satisfied. Given a 
control process together with a standard Wiener process, a solution to Eq. (0) is unique 
if it is indistinguishable from any other solution almost surely satisfying the same initial 
condition. A solution is weakly unique if it has the same law as any other solution with the 
same initial distribution and satisfying Eq. Q for a control process on a possibly different 
stochastic basis so that the joint distributions of control and driving Wiener process are 
the same for both solutions. Let us specify the regularity assumptions to be imposed on 
the coefficients b and a: 

(Al) Cadlag functionals: the mappings 

(^,7) ^ [t^b{iPt,l), t>0], [t^a(Vt), t>0] 

define measurable functionals D^c x F — > D^o and D^o Doo, respectively, where 
L>oo, -Doo are equipped with their Borel cj-algebras. 

(A2) Continuity of the drift coefficient: there is an at most countable subset of [— r, 0], 
denoted by lev, such that for every t > the function defined by 

-Doo X F 9 (^^,7) ^ b{i^t,-f) 
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is continuous on Devif) x T uniformly in 7 G F, where 

Devif) '■= {'0 £ I ip is continuous at t + s for all s £ lev}- 

(A3) Global boundedness: |5|, |cr| are bounded by a constant K > 0. 

(A4) Uniform Lipschitz condition: There is a constant Kl > such that for all ip,(p £ Dq, 
all 7 E r 

15(99,7) - 7)1 + \a{ip) - a{(p)\ < Kl- sup \(p{s)-if{s)\. 

s€[-r,0] 

(A5) Ellipticity of the diffusion coefficient: cr{ip) > ctq for all if G Dq, where cjo > is a 
positive constant. 



Assumptions (i^ and (7^ on the coefficients allow us to invoke Theorem V.7 in IProttei 



( 200,1 : p. 253). which guarantees the existence of a unique solution to Eq. Q for every 
piecewise constant control attaining only finitely many different values. The boundedness 
Assumption (7^ poses no limitation except for the initial conditions, because the state 
evolution will be stopped when the state process leaves a bounded interval. Assumption 
(AlSJ allows us to use "segmentwise approximations" of the solution process, see the proof 
of Proposition ^ The assumptions imposed on the drift coefficient h are satisfied, for 
example, by 

(2) b{(p,-f) := f(^(f{ri),. . . ,ip{rn), j (p{s)wi{s)ds, . . . , j (p{s)w„^{s)ds^ ■ 9{l), 

where ri, . . . ,r„ G [— 0] are fixed, /, g are bounded continuous functions and / is Lip- 
schitz, and the weight functions wi, . . . , Wm lie in L^([— r, 0]). Apart from the control term, 
the diffusion coefficient a may have the same structure as 6 in 

We next give an example of a function that could be taken for a if the cadlag continuity 
in Assumption (A^ were missing. In Section 0] it will become clear that the corresponding 
control problem cannot be approximated by a simple discretisation procedure, because the 
evaluation of cr{if) for any ip £ Dq depends on the discretisation grid. Let Am be the 
subset of the interval [— r, 0] given by 

Am := {(t-2-3^,t] | t = r(^ - 1) for some n e {1, . . . , 2^}}. 

Let A be the union of the sets Am, M £ N. With positive constants o"o, K, we define a 
functional cr : ^ K by 

(3) (T{ip) := ao + K Asup{\ip{t) - (p{t-)\ \ te A}, 

where ip{t—) is the left hand limit of f at t £ [—r,0]. Assumptions (jA|3J and (jA^J are 
clearly satisfied if we choose a according to (jHl , but a would not induce a cadlag functional 
Doo DoQ. This can be seen by considering the mapping [0, 00) 3 t (^{^Pt) for a function 
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tp £ Doo which is constant except for a single discontinuity. If we had defined a with the 
set A being the union of only finitely many sets Am, then we would have obtained a cadlag 
functional. 

We consider control problems in the weak formulation fcf. lYong and Zhoul . ll999l : p. 64). 
Given an admissible control u{.) and a deterministic initial segment 93 G Dq, denote by 
X'^''" the unique solution to Eq. Let I be a compact interval with non-empty interior. 
Define the stopping time of first exit from the interior of / before time T > by 

(4) rj„ := inf{t > | X^'^{t) i int(I)} A f. 

In order to define the costs, we prescribe a cost rate A; : M x T ^ [0, 00) and a boundary 
cost M ^ [0,00) which we take to be (jointly) continuous bounded functions. Let /3 > 
denote the exponential discount rate. Then define the cost functional on Dq x Uad by 

(5) J{^,u) := E (^^\xp(-/3.) • k{X^'^{s),u{s)) ds + 5(^'^'"(r))) , 

where r = t^„. Our aim is to minimize J{ip, .). We introduce the value function 

(6) V{ip) := mi{J{ip, u)\uG Uad}, G Dq. 

The control problem now consists in calculating the function V and finding admissible 
controls that minimize J. Such control processes are called optimal controls or optimal 
strategies. 



3 Existence of optimal strategies 

In the class UnA of admissib le controls it may happen that there is no optimal control (cf. 



Kushner and Dupui^ . 120011 : p. 86). A way out is to enlarge the class of controls, allowing 
for so-called relaxed controls, so that the existence of an optimal (relaxed) control is guar- 
anteed, while the infimum of the costs over the new class coincides with the value function 
V as given by 

A deterministic relaxed control is a positive measure p on BiT x [0,oo)), the Borel 
(T-algebra on F x [0,oo), such that 

(7) p{T X [0,t]) = t for alH > 0. 

For each G S B(r), the function t ^ p{G x [0, t]) is absolutely continuous with respect 
to Lebesgue measure on [0, 00) by virtue of property Denote by p{., G) any Lebesgue 
density of p{G x [0, .]). The family of densities p{.,G), G £ B{T), can be chosen in a Borel 
measurable way such that p{t, .) is a probability measure on B(T) for each t > 0, and 

p{B) = / / l{(-,,t)6B} p(t, d-i) dt for all B£B{Tx [0, 00)). 
JO Jv 
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Denote by TZ the space of deterministic relaxed controls which is equipped with the weak- 
compact topology induced by the following notion of convergence: a sequence (pn)neN of 
relaxed controls converges to p S 7^ if and only if 

j g{-f,t)dpn{j,t) ''-^ j g{j,t)dpij,t) for all 5 G Cc(r x [0, cx))), 

Fx [0,00) Fx [0,00) 

where Cc(r x [0, 00)) is the space of all real- valued continuous functions on F x [0, 00) having 
compact support. Under the weak-compact topology, TZ is a (sequentially) compact space. 

Suppose {pn)n£N is a convergent sequence in TZ with limit p. Given T > 0, let Pn\T 
denote the restriction of pn to the Borel a-algebra on T x [0,T], and denote by p^x the 
restriction of p to B{T x [0,T]). Then Pn\T^ n E N, p^x are all finite measures and 
converges weakly to p\x- 

A relaxed control process is an 7?.- valued random variable R such that the mapping 
cj R{r X [0, t])(LL') is .Ff-measurable for all t > 0, G € 13{T). For a relaxed control 
process R Eq. Q takes on the form 

(8) dX{t) = (^j b{Xt,^)R{t,d^)yt + a{Xt)dW{t), t>0, 

where {R{t, .))t>o is the family of derivative measures associated wit h R. The family 
{R{t, .)) can be constructed in a measurable way fcf. iKushnen . Il99(l : p. 52). A relaxed 
control process together with its stochastic basis including the Wiener process is called 
admissible relaxed control if, for every deterministic initial condition, Eq. ((HJ has a unique 
solution which is also weakly unique. Any ordinary control process u can be represented 
as a relaxed control process by setting 

R{B) := / / <5„(t)(d7) dt, B G B{T x [0, 00)), 

Jo Jr 

where 6^ is the Dirac measure at 7 G F. Denote by Uad the set of all admissible relaxed 
controls. Instead of (EJ we define a cost functional on Dq x U^i^ by 

(9) J{ip,R) := eI^J^ J^eM-(3s)-k{X^'^{s),-/)Ris,d-f)ds + ^(^^'^(t))) , 

where X*^'^ is the solution to Eq. ^ under the relaxed control process R with initial 
segment 99 and r is defined in analogy to Q. Instead of ^ as value function we have 

(10) V{^) := inf{ J(99, R) \ R £ Uad}, ^ G Dq. 

The cost functional J depends only on the joint distribution of the solution X^'^ and the 
underlying control process R, since r, the time horizon, is a deterministic function of the 
solution. The distribution of X'^'^, in turn, is determined by the initial condition 99 and 
the joint distribution of the control process and its accompanying Wiener process. Letting 
the time horizon vary, we may regard J as a function of the law of {X, R, W, r), that is, as 
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being defined on a subset of the set of probability measures on B{Doo xTZx D^o x [0, oo]). 
Notice that the time interval has been compactified. The domain of definition of J is 
determined by the class of admissible relaxed controls for Eq. ((HI), the definition of the 
time horizon and the distributions of the initial segments Xq. 

The idea in proving existence of an optimal strategy is to check that J{^p, .) is a (se- 
quentially) lower semi-continuous function defined on a (sequentially) compact set. It then 
follows from a theorem bv Weierstra£ that J( (p, .) attains its minimum at some point of 
its compact domain (cf. lYong and Zhou ^ 1999, : p. 65) . Th e following proposition gives the 
analogue of Theorem 10.1.1 in lKushner and DupuiFl (|2nnil : pp. 271-275) for our setting. We 
present the proof in detail, because the identification of the limit process is different from 
the classical case. 



Proposition 1. Assume (j^) - (j^^- Let {{R^ ,W^^))MeN be any sequence of admis- 
sible relaxed controls for Eq. where {R'^ jW'^) is defined on the filtered probability 
space {SIm,J^^\{H^),^m)- Let X^^ he a solution to Eq. under control {R^\W^) 
with deterministic initial condition ip^^ G Dq, and assume that ((/J^^) tends to ip uni- 
formly for some ip G Dq. For each M E N, let be an {!F^)- stopping time. Then 
((X^^i^*^VF^^r*0)AfeN tight. 

Denote by {X,R,W,t) a limit point of the sequence {{X^\ R^^ ,W^\T^^))Mm- Define 
a filtration by Tt := a(X(s), ii(s), l^(s), rl{^<i}, s < t), t > 0. Then W{) is an {Tt)- 
adapted Wiener process, r is an (J^t)- stopping time, {R, W) is an admissible relaxed control, 
and X is a solution to Eq. ^ under {R,W) with initial condition (f. 



Proof. Tightness of (X*^) follows from the Aldous criterion fcf. iBillingslevl . Il999l : pp. 176- 
179): given M G N, any bounded (^*^)-stopping time u and S > we have 



E 



Ml 



\X^\iy + 6)- X^^{i^)\'^ I ) < 2K^S{S + 1) 



as a consequence of Assumption and the Ito isometry. Notice that X*^(0) tends 

to X{0) as M goes to infinity by hypothesis. The sequences {R^'^) and (r^^) are tight, 
because the value spaces TZ and [0,oo], respectively, are compact. The sequence {W'^^) is 
tight, since all VF*^ induce the same measur e. Finally, componentwise tightness implies 
tightness of the product (cf. IBillingslevl . Il999l : p. 65). 

By abuse of notation, we do not distinguish between the convergent subsequence and the 
original sequence and assume that {{X'^ , R^ , .,t^)) converges weakly to {X, R, W, r). 
The random time r is an (.Ft)-stopping time by construction of the filtration. Likewise, 
R is (jri)-adapted by construction, and it is indeed a relaxed control process, because 
R{T X [0,t]) = t, t > 0, P-almost surely by weak convergence of the relaxed control 
processes {R^'^) to R. The process W has Wiener distribution and continuous paths with 
probability one, being the limit of standard Wiener processes. To check that W is an 
{J-t)-Wienei process, we use the martingale problem characterization of Brownian motion. 
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To this end, for g G Cc(r x [0, oo)), p gTZ define the pairing 



ig,p)it) ■■= / gi^,s)dpi^,s), t>0. 
Jrx[o,t] 

Notice that real-valued continuous functions on TZ can be approximated by functions of 
the form 

7^ 3 H{{gj,p){ti), e Np X N,) G M, 

where p, q are natural numbers, {ti \ i G Np} C [0, oo), and H, gj, j G Ng, are suitable 
continuous functions with compact support and Nat := {!,... , N} for any N £ N. Let 
t > 0, ti,...,tp G [0,t], h > 0, gi,...,gq be functions in Cc{T x [0, cxd)), and H he a 
continuous function of 2p + p-q + 1 arguments with compact support. Since W'^ is an 
-Wiener process for each M G N, we have for all / G Cc(M) 



0. 



EM[H{X^'{ti), {gj,R^'){ti),W^'{ti),T^'l^,M<t}, G Np X N,) 

t+h 

• [f{W^{t + h)) - f{W^\t)) - \ I ^{W''{s))ds)) = 

t 

By the weak convergence of {{X^'^ , R^'^ ,W^\t^'^))m& to {X,W,R,t) we see that 
E(^H{X{U),igj,R){U),Witi),Tl{,^ty, (ij) G Np x N^) 

t+h 

{t + h))-f{W{t))-^- I ^{W{s))ds 



for all / G C^(M). As H, p, q, ti^ gj vary over all possibilities, the corresponding random 
variables H{X{ti),{gj,R){ti),W{ti),Tl^^<^fy, G Np x Ng) induce the ci-algebra J^f 

Since t > 0, h > were arbitrary, it follows that 

i 

f{W{t)) - f{W{0)) -\j ^{W{s))ds, t > 0, 



is an (^t)-martingale for every / G C2(M). Consequently, W is an (^t)-Wiener process. 

It remains to show that X solves Eq. ^ under control {R, W) with initial condition 
if. Notice that X has continuous paths on [0, oo) P-almost surely, because the process 
{X{t))t>o is the weak limit in Doo of continuous processes. Fix T > 0. We have to check 
that P-almost surely 

X{t)=^{0)+ [ [ b{Xs,-f)R{s,dj)ds+ [ (T(X,)dP^(s) for alH G [0,T]. 
Jo Jr Jo 



By virtue of the Skorohod representation theorem (cf. iBillingsleyl . p. 70) we may 

assume that the processes (X*^, R^^ , W^), M G N, are all defined on the same probability 
space (0,J^,P) as {X,R,W) and that convergence of {{X^^ , R^'' ,W^^)) to {X,R,W) is 
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P-almost sure. Since X, W have continuous paths on [0,T] and (9?^) converges to tp in 
the uniform topology, one finds Q. ^ !F with P(r2) = f such that for all a; G $7 

sup \X^^^ {t){uj) - X{t){oj)\ 0, sup \W^\t){oj)-W{t){oj)\ *^ 0, 

te[-r,T] te[-r,T] 

and also R'^^{uj) R{uj) in TZ. Let G 0. We first show that 

/ H,7)i?*^(.,d7)Hds r / 6(X,(a;),7)i?(.,(i7)Mds 

Jr Jo Jr 

uniformly in t G [0, T]. As a consequence of Assumption (i^, the uniform convergence of 
the trajectories on [— r, T] and property of the relaxed controls, we have 



/ M,7) -6(X,H,7)|(ii?*^(7,5)M 

Jrx[o,T] 



0. 



By Assumption (i^, we find a countable set C [0,T] such that the mapping (7,5) 1— > 
b{Xs{uj),j) is continuous in all (7,5) G T x ([0,T] \ A^^). Since A,., is countable w e have 



R{uj){r X Atj) = 0. Hence, by the generalized mapping theorem fcf. lBillingslevl . ll999i : p. 21) 
we obtain for each t G [0, T] 



b{Xs{uj),j)dR^'{j,s){Lo) "^^"^ / 6(X,H,7)di?(7,s)H. 

rx[o,t] Jrx[o,t] 

The convergence is again uniform in t G [0,T], as 6 is bounded and R^, M eN, R are 
all positive measures with mass T on F x [0, T]. Define cadlag processes C^, M G N, on 
[0,00) by 

C*^(t) := (^*^(0) + / 6(Xf,7)di?*'^(7,s), t>0, 
Jrx[o,t] 

and define C in analogy to C^^ with if, R, X in place of R^^ , X*^, respectively. Prom 
the above, we know that C^\t) C(t) holds uniformly over t G [0,T] for any T > with 
probability one. Define operators F*^ : I)oo ^ -Doo, M G N, mapping cadlag processes to 
cadlag processes by 

f (Y(t+s)(io) ift+s>0,\ 

y [(^^^(t+s) else J 

and define F in the same way as with y?*^ replaced by (p. Observe that X^'^ solves 

X^{t) = C^{t) + f F^{X^){s-)dW^{s), t>Q. 
Jo 

Denote by {X{t))t>o the unique solution to 

X{t) = C{t) + / F{X){s-)dW{s), t>0, 
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and set X{t) := (p{t) for t G [— r, 0). Assumption and the uniform convergence of 

to 99 imply that F*^(X) converg es to FiX) un iformly on compacts in probability 



(convergence in ucp). Theorem V.15 in 
to X in ucp, that is 



Protted ()2nn.1 : p. 265) yields that {X^) converges 



sup \X''\t) 
te[o,r] 



X{t)\ 



in probability P for any T > 0. 



Therefore, X is indistinguishable from X. By definition of C and F, this implies that X 
solves Eq. ^ under control {R,W) with initial condition if, and so does X. □ 

If the time horizon were deterministic, then the existence of optimal strategies in the 
class of relaxed controls would be clear. Given an initial condition if G Dq, one would 
select a sequence {{R^^,W^))MeN such that (J(^,i?*^)) converges to its infimum. By 
Proposition^ a suitable subsequence of {{R^ , W'^)) and the associated solution processes 
would converge weakly to {R,W) and the associated solution to Eq. Q. Taking into 
account (jHl, the definition of the costs, this in turn would imply that J{^,-) attains its 
minimum value at R or, more precisely, {X,R,W). 

A similar argument is still valid, if the time horizon depends continuously on the paths 
with probability one under every possible solution. That is to say, the mapping 

(11) t: Doo^ [0,00], f(V') := inf{t > | V(i) ^ mt(/)} A f, 

is Skorohod continuous with probability one under the measure induced by any solution 
X^'^, R any relaxed control. This is indeed the case if the diffusion coefficient a is bounded 
away from zero as required by Assumption {J^, cf. iKushner and Dupuid pOOll : pp. 277- 
281). 

By introducing relaxed controls, we have enlarged the class of possible strategies. The 
infimum of the costs, however, remains the same for the new class. This is a consequence of 
the fact that stochastic relaxed controls can be arbitrarily well approximated by piecewise 
constant ordinary stochastic controls whic h attain onlv a f inite number of different control 
values. A proof of this assertion is given in Kushner pp. 59-60) in case the time hori- 



zon is finite, and extended to the case of control up to an exit time in lKushner and Dupui 



( 200ll : pp. 282-286). Notice that nothing hinges on the presence or absence of delay in the 
controlled dynamics. Let us summarize our findings. 

Theorem 1. Assume - (J^^. Given any deterministic initial condition (p £ Dq, the 
relaxed control problem determined by ^ and ^ possesses an optimal strategy, and the 
minimal costs are the same as for the original control problem. 



4 Approximating chains 

In order to construct finite-dimensional approximations to our control problem, we discre- 
tise time and state space. Denote by /i > the mesh size of an equidistant time discretisa- 
tion starting at zero. Let Sh '■= ^/KL be the corresponding state space, and set 1^ := ICiSh. 
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Notice that Sh is countable and Ih is finite. Let A/j : M ^ S*/! be a round-off function. We 
will simplify things even further by considering only mesh sizes h = jj for some M G N, 
where r is the delay length. The number M will be referred to as discretisation degree. 

The admissible controls for the finite-dimensional control problems correspond to piece- 
wise constant processes in continuous time. A time-discrete process u = (ti(n))neNo on a 
stochastic basis (^t), P) with values in F is a discrete admissible control of degree 

M if u takes on only finitely many different values in F and u{n) is ^nft-nieasurable for all 
n € Nq. Denote by {u{t))t>o the piecewise constant cadlag interpolation to u on the time 
grid. We call a time-discrete process (^(ra))„g|_jv/,...,o}uN a discrete chain of degree M if 
(^(n)) takes its values in Sh and ^(n) is ^„/i-nieasurable for all n £ Nq. In analogy to u, 
write {(^{t))t>-r for the cadlag interpolation to the discrete chain (C(^))ne{-Af,...,o}uN- We 
denote by the Dq- valued segment of ^(.) at time t > 0. 

Let if £ Dq he a deterministic initial condition, and suppose we are given a sequence of 
discrete admissible controls {u^)m&n, that is u^^ is a discrete admissible control of degree 
M on a stochastic basis {QM,J^^,{^t^)^^M) for each M € N. In addition, suppose 
that the sequence (u*^) of interpolated discrete controls converges weakly to some relaxed 
control R. We are then looking for a sequence approximating the solution X of Eq. ^ 
under control {R, W) with initial condition if, where the Wiener process W has to be 
constructed from the approximating sequence. 

Given M-step or extended Markov transition functions p^'^ : S^^'^'^ x F x 5"^ ^ [0, 1] , 
M € N, we define a sequence of approximating chains associated with 93 and {u^'^) as a 
family (^^'^)jvfeN of processes such that is a discrete chain of degree M defined on the 
same stochastic basis as u^^ , provided the following conditions are fulfilled for h = hM '■= jj 
tending to zero: 

(i) Initial condition: ^*^(n) = Ah{(p{nh)) for all n G {-M, ... ,0}. 

(ii) Extended Markov property: for all n S Nq, all a; E S*/! 

PM(e^(n + l)=x|.F*^) = /^(e*^(n-M),...,e*^(n),u^(n),x). 

(iii) Local consistency with the drift coefficient: 

/z^mH := EM(e'"(n+l) - e\n) \ T^y) 

= h-b{C::i,u''{n))+oih) =: h-hH{^Z^"\n)). 

(iv) Local consistency with the diffusion coefficient: 

^M{{e\n+l)-e'{n)-^ieAn)f\T^h) = h ■ a^^^^) + o{h) =:h-aU^l). 

(v) Jump heights: there is a positive number iV, independent of M, such that 

sup l^Hn + 1) - e*'(n)| < NVh^. 
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It is straightforward, under Assumptions (jAIH} and (AlHl, to construct a sequence of ex- 
tended Markov transition functions such that the jump height and the local consistency 
conditions can be fulfilled. Assuming that the bounding constant K from is a natural 
number, we may define the functions for all M G N big enough by, for example, 

kcj(Z) + ||&(Z,7), {{x = Z{{))+KVh, 



p^^(Z(-M),...,Z(0),7,x) := 



^a{Z) - ^b{Z, 7), if X = Z(0) - KVh, 
l-^a{Z) ifa; = Z(0) 

else, 



where h = hu, Z = (Z(-M), . . . , Z(0)) G 5^+^ 7 e T, x e 5,,, and Z £ Dq is the 
piecewise constant interpolation associated with Z. The family {p^^) as just defined, in 
turn, is all we need in order to construct a sequence of approximating chains associated 
with any given (u*^). 

We will represent the interpolation as a solution to an equation corresponding 
to Eq. (Pi with control process and initial condition 93*'^, where (p^ is the piecewise 
constant 5/i-valued cadlag interpolation to ip, that is (/?*^ = ^g^. Define the discrete process 
{L^\n))nm, by L*^(0):=Oand 

n-l 
1=0 

Observe that is a martingale in discrete time with respect to the filtration (^^). 
Setting 



ef{t):= E h.h,,{i^,v^^{iKj) - [ b{^f,u^'\s))ds, t>0, 

the interpolated process can be represented as solution to 

^-^(t) = ^Af(o) ^ f\{^f,u^is))ds + L^^(LIJ) + sf(i), t>0. 

Jo 



With T > 0, we have for the error term 



Ea/ sup,e[o,r]kfW| < E hBM(\bH{^fh,u^'{i))-b{^^;i,u^{^))\) + K-h 



i=0 



which tends to zero as M goes to infinity by Assumptions (j^, (^AEJ, dominated conver- 
gence and the defining properties of (^*^). Moreover, |e]^(t)| is bounded by 2K-T for all 
t E [0, T] and all M big enough, whence also 



-m ( I M/.\|2\ M^oo „ 

EM(^suptg[o,T]|ei j — > 0. 
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The discrete-time martingale can be rewritten as a discrete stochastic integral. Define 

{W^Hn))neNo by setting W^^{0) := and 

n— 1 ^ 

Using the piecewise constant interpolation W^^ of T^*^, the process can be expressed 
as the solution to 



(12) eHt) = /'(O) + ds + dW'Hs) + (t), t > 



10 Jo 

where the error terms (£2^) converge to zero as (e*^) before. 

We a re now prepared for the conv ergence result, which should be compared to Theorem 
10.4.1 in 



Kushner and Dupuia l|2nflll : p. 290). The proof is similar to that of Proposition ^ 



We merely point out the main differences. 

Proposition 2. Assume - (J^. For each M £ N, let he a stopping time with 
respect to the filtration generated by {^'^\s),u'^^ {s),W'^\s), s < t). Let i?*^ denote the 
relaxed control representation of u^^ . Suppose (v?*^) converges to the initial condition ip 
uniformly on [-r,0]. Then {{l^'^ , R^\W^\T^^))Mm is tight. 

For a limit point {X,R,W,t) set J^t ■= cr{X{s), R{s),W{s),Tl{^<ty, s < t), t > 0. 
Then W is an (J^t)- adapted Wiener process, r is an (J^t) stopping time, {R,W) is an 
admissible relaxed control, and X is a solution to Eq. ^ under {R, W) with initial condition 
<P- 

Proof. The main differences in the proof are establishing the tightness of (W^) and the 
identification of the limit points. We calculate the order of convergence for the discrete-time 
previsible quadratic variations of {W^^): 

n— 1 n— 1 

{W^)n = Y.B{iW'H^ + l)-W'\^)f\^^^) = nh + o{h)Y: 



1=0 



'^'^h) 



for all M G N, n S Nq. Taking into account Assumption and the definition of the 

time-continuous processes , we see that {W^'^) te nds to Id^n rv^^ in probability uniformly 



on compact time intervals. By Theorem VIII.3.11 of l.Tacod and ShirvaevI (|l987l : p. 432) we 
conclude that {W^^) converges weakly in to a standard Wiener process W . That W 
has independent increments with respect to the filtration {J^t) can be seen by considering 
the first and second conditional moments of the increments of for each M G N and 
applying the conditions on local consistency and the jump heights of (^^^). 

Suppose {{i^ ,R^\W^^)) is weakly convergent with limit point {X,R,W). The re- 
maining different part is the identification of X as a solution to Eq. ^ under the relaxed 
control (i?, W) with initial condition ^p. Notice that X is continuous on [0, 00) because 
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of the condition on the jump heights of (C^^), cf. Theorem 3.10.2 in lEthier and Kurtz 
( 198(1 : p. 148). Let us define cadlag processes C*^, C on [0, oo) by 



JO 



C{t) := (^(0) + / b{Xsn)dR{s,j), 
irx[o,t] 



t > 0, 
t > 0. 



Then C, C*^ are bounded on compact time intervals uniformly in M S N. Invoking 
Skorohod's representation theorem, one establishes weak convergence of (C^^) to C as in 
the proof of Proposition ^ 

The sequence (W^) is of uniformly controlled variations, hence a good sequence of in- 
tegrators in the sense of lKnrtz and Protten l|l991l l. because the jump heights are uniformly 
bounded and is a martingale for each M £ N. We have weak convergence of (14^*^) 
to W. The results in lKurtz and Protten (|l99lh guarantee weak convergence of the corre- 
sponding adapted quadratic variation processes, that is ([1^*^,1^*^]) converges weakly to 
[W, VF] in Doo = -Dr([0,oo)), where the square brackets indicate the adapted quadratic 
(co-)variation. Convergence also h olds for the sequenc e of pr ocess pairs {W^ , [W^ , W^^]) 



in -DiR2([0, oo)), see Theorem 36 in lKurtz and Protten l|2fln4r i. 

We now know that each of the sequences (C^), (C^), {W^), {[W^^,W^]) is weakly 
convergent in -D]r([0, oo)). Actually, we have weak convergence for the sequence of pro- 
cess quadruples {^^'^ ,W^'^ ,[W^^ ,W^^]) in L>]g4([0, oo)). To see this notice that each 
of the sequences (p^ + C^^), (p^ + W^^), (^^^ + [W^^,W^^]), (C^ + VF*^, (C^'^ + 
[W^,W^]), and {W^^ + [W^\W^^]) is tight in Z)k([0,oo)), because the limit processes 



e x. W. and \W.W 



m 



/dro oo) are all continuous on [0,oo). According to Problem 22 



Ethier and Kurtzl l|198(il : p. 153) this implies tightness of the quadruple sequence in 



L']r4([0, oo)). Since the four component sequences are all weakly convergent, the four- 
dimensional sequence must have a unique limit point, namely (X, C, W, [VF, 11^] ) . By 
virtue of Skorohod's theorem, we may again work under P-almost sure convergence. Since 
C, X, W, [W,W] are all continuous, it follows that C*^ C, ^ X, W^^ W, 
[W^ jW'^^] —>■ [W,W] uniformly on compact subintervals of [0, oo) with probability one. 
Define the mapping F: Dq x D^o D^o by 



F{^,x){t) := a [-r,0]3s 



x{t+s) ift+s>0, 
ip{t+s) else 



t > 0. 



For M G N, let be the mapping from -Dqo to L>oo given by F 



Ml 



F{ip^\x). Let 



H^^ : Dqo — ^ -Coo be the cadlag interpolation operator of degree M, that is H^{x) is the 
piecewise constant cadlag interpolation to x G -Dqo along the time grid of mesh size 
starting at zero. Define F*^ : l)oo -C'oo by 



F^[x){t):= F{^^\H^{x)){yt\M), t>0, 
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where [t\M ■= i[[ft\. If -0 G -Doo, we will take F^(V'), F^(V') and F(V') to equal 
F^{x), F^'\x) and F{ip,x), respectively, where x is the restriction ofip to [0,oo). Eq. (|T2|l 
translates to 

f = C^'(t) + f F^\e'){s-)dW^'{s), t>0. 
Jo 

Let ^ be the unique cadlag process solving 

i{s) = ^{s), se[-r,0), i{t) = C{t) + f F{i){s-)dW{s), t>0. 

Jo 

Fix r > 0. Since converges to X as M goes to infinity uniformly on compacts with 
probability one, it is enough to show that 

(*) e( sup \m-eHt)\') ''-^ 0. 

First observe that 

e( sup \C{t) - C'\tf) 0, sup \m - e\tf 0, 

because C is uniformly bounded on compact time intervals and is cadlag and continuous 
on [— r, 0). Given e > 0, by Lemma ^ in the Appendix and by Gronwall's lemma we find 
that there is a positive number Mq = Mo(e) such that for all M > Mq 

e( sup f F{i){s-)dW{s)- f F^'\i^^){s-)dW^'{s) ^) < 76Te{K^+l)exp{4KlT). 
He[o,T] Jo Jo ' 

This yields Q and the assertion follows. □ 

If we consider approximations along all equidistant partitions of [— r, 0] , then the hy- 
pothesis about the uniform convergence of the initial conditions implies that ip must be 
continuous on [— r, 0] \{0}. In case has jumps at positions locatable on one of the equidis- 
tant partitions, the convergence results continue to hold when we restrict to a sequence of 
refining partitions. 



5 Convergence of the minimal costs 

The objective behind the introduction of sequences of approximating chains was to obtain 
a device for approximating the value function V of the original problem. The idea now is to 
define, for each discretisation degree M G N, a discrete control problem with cost functional 
so that J*^ is an approximation of the cost functional J of the original problem in 
the following sense: Given a suitable initial segment p ^ Dq and a sequence of discrete 
admissible controls (u*^) such that {u^^) weakly converges to a relaxed control R, we have 
J{ip,u'^'^) —>■ J{ip,R) as M tends to infinity. Under the assumptions introduced above, 
it will follow that also the value functions associated with the discrete cost functionals 
converge to the value function of the original problem. 
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Fix M G N, and let h:= jj. Denote by the set of discrete admissible controls of 
degree M. Define the cost functional of degree M by 



(13) J^'{v,u) := E ( ^ expi-Pnh) ■ A;(^(n), ^x(n)) • h + g{aNh)) 

\ n=0 

where ip G Dq, u G U^^ is defined on the stochastic basis (i7, (JT^ ), P) and {i{n)) is a 
discrete chain of degree M defined according to p^^ and u with initial condition Lp. The 
discrete exit time step is given by 

(14) Nh := mm{n G No | C(n) ^ 4} A [fj- 

Denote by r*^ := /i • A'^/j the exit time for the corresponding interpolated processes. The 
value function of degree M is defined as 

(15) V^ip) := mf{j''{^,u) \ u G U^',], p G D^. 

We are now in a position to state the result about convergence of the mi nimal costs. Propo- 
sitionBl and Theorem|2lare comparable to Theorems 10.5.1 and 10.5.2 in lKnshner and Dupuis 
(|200ll : pp. 292-295). Let us suppose that the initial condition ip £ Dq and the sequence of 
partitions of [— r, 0] are such that the discretised initial conditions converge to p uniformly 
on [— r, 0]. 

Proposition 3. Assume (^-(^. If the sequence {^^ ,f^) of interpolated 
processes converges weakly to a limit point {X, R,W,t) , then X is a solution to Eq. ^ 
under relaxed control {R, W) with initial condition p, r is the exit time for X as given by 
(@J, and we have 

Proof. The convergence assertion for the costs is a consequence of Proposition the fact 
that, by virtue of Assumption (i^, the exit time f defined in ifTTIl is Skorohod-continuous, 
and the definition of and J (or J). □ 

Theorem 2. Assume 

(4W - Then we have limAf^oo V^{p) = V{ip). 

Proof. First notice that liminfjv/_^oo ^*^(v5) > ^{^) as a consequence of Propositions [21 
and ini In order to show limsupjv,/_^o^ y^^((^) < V{p) choose a relaxed control {R,W) 
so that J((/3, R) = V[p) according to Proposition ^ Given e > 0, one can construct a 
sequence of discrete admissible controls {u^^) such that ((^^"^j -u^^, ly*'^, f*^)) is weakly 
converg ent, where (^*^), (H^*^), (r^) are constructed as above, and 

lim sup I J^' {if, u^^) - J {if, R)\<e. 

M^oo 

The existence of such a sequence of discrete admissible controls is guaranteed, cf. the 
discussion at the end of Section |31 By definition, V'^^{ip) < J^'\(p,u^^) for each M G N. 
Using Proposition 01 we find that 

limsupy^^(V9) < limsup J*^((^,u*^) < V{ip)+e, 
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and since e was arbitrary, the assertion follows. □ 

Appendix 

The proof of the following lemma makes use of standard tech niques . In t he context of 
approximation of SDDEs, it should be compared to Section 7 in iMad 1120031 ). 

Lemma 1. In the notation and under the assumptions of Proposition\^it holds that for 
every e > there is Mq G N such that for all M > Mq 

rt rt 



e( sup \[ F{0{s-)dW{s) - [ F^{^^^){s-)dW^^{s)f 

^tG[0,T] Jo Jo 

< AKl / E( sup \i{t) - i^' {t)\^) ds + 76TeiK^ + l). 

Jo H6f-r,sl ^ 



Proof. Clearly, 



Ef sup \ [' F{i){s-)dW{s)- I F*^(r-')(s-)dVF^"'(s)| 

^t6[0,T] Jo Jo 

ft ft 



(16) < 2e(sup|/ F{i){s-)dW{s)- [ F^\C^^){s-)dW{s 

\elo,T] Jo Jo 

+ 2e( sup I f F^^{i^^^){s-)dW{s)- f F'"\i^'){s-)dW^{s 
He[o,T] Jo Jo 

Using Doob's maximal inequality, Ito's isometry, Fubini's theorem and Assumption (jA^, 
for the first expectation on the right hand side of (|T6|l we obtain the estimate 

e( sup I f F{i){s-)dW{s)- f F^'{i^'){s-)dW{s)\^) 
He[0,T] Jo Jo ' 

(17) < 4^^E(|F(|)(s)-F*^(f'0(s)f)ds 
< 4Kl f sup |e(t)-e*'(t)|')ds. 

JO VG[-r,s] ' 

Fix any G N. The second expectation on the right hand side of (fT6)l splits up into three 
terms according to 

ft rt 



e( sup I / F^{^^^'){s-)dW{s)- [ F^^{^^){s-)dW^^{s)\ 
H6[0,r] Jo Jo 

< 4e( sup I f F^\i^'){s-)dW{s)- f F^{i^'^){s-)dW{s)\^ 
He[0,T] Jo Jo 

+ 4e( sup I f F^{i^^){s-)dW{s)- f F^{i^"){s-)dW^"{s) 
He[0,T] Jo Jo 

+ 4e( sup I f F^{C'^'^){s-)dW{s)- [' F''\e'){s-)dW^'{s)f). 
HefO.Tl Jo Jo ' 
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Again using D oob's maximal inequality and a generalized version of Ito's isometry (cf. 



Protten . 1200,11 : pp. 73-77), for the first and third expectation on the right hand side of (|T8|l 



we get 



(19) 



and 



(20) 



E( sup 1/ F^^{^^'^){s-)dW{s)- I F''{C'''){s-)dW{s) 



te[o,T] Jo 

T 



< 4 E ( / \F^' (f ^) (s) - F'' ){s)\Us 



N /7M\ 







e( sup I / F''{e'){s-)dW^'{s)- [ F^\e')is-)dW^\s)\ 
He[o,T] Jo Jo 

< iB(^£\F'^\e'){s-)-F^ie')is-)\^d[W^,W''']is) 



respectively. Notice that, path-by-path, we have 

'"'^|F^^(C"^)(s-) - F^ie')is-f d[W^', W^'] is) 







i=0 



In order to estimate the second expectation on the right hand side of (fTH|l . observe that, 
P- almost surely, for all t £ [0, T] 

F^{e')is-)dW{s) = F^(C*')(LtJiv) • (wit) - W{[t\r,)) 



i=0 



as F ) is piecewise constant on the grid of mesh size On the other hand, 
j'^F^{i^){s-)dW''{s) = F^{^''){[t\^)-{w^'{t)-W^{[t\j,)) 



+ E F''{e'){j,^)■{w''{j,{^+l))-W''{ 



N' 



i=0 



By Assumption (AOJ, \a\ is bounded by a constant K, hence 



F''{i^){s-)dW{s)- I F'' {i^'){s-)dW^'\s) 



rM I 



< 2Kl^t\ ■ sup \W{s) -^'{3)1 < 2KfT- SUV \W{s)-W''''{s)\. 



se[o,t] 



se[o,T] 
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Bounded convergence yields for each fixed G 



(21) E sup 



tM, 



Let x,y £ Doo- By Assumption we have for all t £ [0,T] 

|F^(y)(t)-F(v.,x)(t)| = |F((/,^,i/^(y))(LtJ;v)-F(^,x)(t)| 



< Kl- sup \ip'\s) - ^is)\ + Kl- sup \H'^y){s) - xis)\ 

s6[-r,0) se[0,T] 

+ \F{ip,x){[t\N)-F{ip,x)it)\. 

By Assumption (^A^, the map [0, T] 9 t F{ip,x){t) is cadlag, whence it has only finitely 
many jumps larger than any given positive lower bound. Thus, given e > 0, there is a 
finite subset A = A[e,T,ip,x) C [0,T] such that 

limsup |F((^,x)([tj7v) - <e for all t G [0, T] \ A. 



Moreover, the convergence is uniform in the following sense fcf. iBillingsleyl . Il993 ) : We can 
choose the finite set A in such a way that there is A^'o = NQ{e, T, ip,x) so that 

\F{ip,x){[t\N) - F{ip,x){t)\ < 2e for alH G [0,r] > iVo. 

Given e > 0, we therefore find G N and an event (l with P(i^) > 1 — e so that for 
each uj £ (l there is a finite subset A^ C [0, T] with < Ne and such that for all 

t G [0, T] \ A^ and all M > A^ we have 

|F^'^(e*^H)(t)-F(XH)(t)f + |F^(f^H)(t)-F(XH)(t)p < e. 

The expression on the right hand side of (fT9|l is then bounded from above by 9Te{K'^ + 1). 
For M big enough, also the expression on the right hand side of IpT]) is smaller than 
9Te{K'^ + 1), and the expectation in ipTjl is smaller than Te. □ 
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